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Executive Summary 



Dr. Jo Lynne DeMary, Virginia’s state superintendent of public instruction, requested 
that the Appalachia Educational Laboratory at Edvantia work in partnership with the Virginia 
Department of Education and Petersburg City Schools to design and test the Partnership for 
Achieving Successful Schools Initiative (PA+SS) Model IV Intervention. The goal was to help 
the schools in Petersburg move toward full accreditation in the state’s accountability system. 
Unlike three other intervention models being tested in Virginia, Model IV called for placing one 
school improvement specialist in each of the division’s 10 schools and another school 
improvement specialist in the central office. Of the four models, only Model IV involves 
working at the school division’s central office. Edvantia selected The CNA Corporation (CNAC) 
to conduct a formative and summative evaluation of the Model IV Intervention implementation 
and impact. 

This report presents the final evaluation findings for the Model IV Intervention 
implemented in the Petersburg City Schools Division during school years 2002-2003 through 
2004-2005 of the 3-year divisionwide school improvement project. The report presents an 
analysis of the fidelity with which the intervention was implemented. It explores the role and 
effectiveness of the school improvement specialists; the collaboration among the division’s 
central office, the schools, and school improvement specialists; the completeness of each 
school’s School Improvement Plan (SIP); and the changes that have occurred in school climate 
and capacity for improvement. Included is a comparative analysis of the percentages of students 
passing the Standards of Learning (SOL) tests in Petersburg and comparison schools during the 
four years prior to the Model IV Intervention and the two years for which student achievement 
data are available since the intervention was initiated.* 

Findings 

Overall findings at the end of Year 3 are mixed. On the one hand, implementation of the 
Model IV Intervention in Petersburg schools was achieved through the dedication of the school 
improvement specialists, the central office specialist, the schools’ instructional specialists, and 
the school leaders’ efforts to collaborate and build school capacity. These efforts were enhanced 
by the central office specialist during the third year with the creation of the Central Office 
Support Team (COST). The schools’ instructional specialists also were trained in data analysis 
and coached by the improvement specialists to take leadership roles in the reform effort. On the 
Continuous School Improvement Questionnaire (CSIQ), which revealed professional staffs’ 
perceptual evidence of school improvement, staff reported improvement on all six components 
measured by the CSIQ over the course of the intervention. An assessment of each school 
improvement plan (SIP) provided further evidence that the intervention components focused on 
improving the planning process were implemented with fidelity throughout the division. Because 
Standards of Learning (SOL) test scores were not yet available from the third year of the 
intervention, evidence about the long-term impact on student achievement as measured by the 
SOL exams is inconclusive. However, SOL data from Years 1 and 2 of the intervention show an 
overall increase in pass rates for the majority of students at the elementary, middle, and high 



Student achievement data from the third year of the intervention were not yet available at the time this report was 
written. Therefore, evaluation data include 3 years of all data except student achievement outcomes, for which 
there are data from the first two years of the intervention. 
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school levels. A trend line analysis shows that most of these pass rates are increasing at a greater 
rate since the implementation of the program. The increasingly rapid rise in pass rates during the 
study period suggests that the PA+SS Model IV Intervention may be having a positive impact on 
school improvement. 

On the other hand, the Model IV Intervention appeared to have been hampered by a 
school board that did not appear to stand united in its support of the intervention. The instability 
of leadership during the study period (there were three different superintendents in 3 years) also 
might have hindered implementation fidelity, especially during the first year. Staff in the schools 
perceived the schools’ capacity for improvement to decrease or remain unchanged during the 
study period. In addition, there are some endemic issues that plague the Petersburg City Schools 
Division such as high professional turnover rates, low numbers of highly qualified teachers, 
difficulties in instituting systemic change, and a perceived continuing lack of full collaboration 
between the central office staff and school staff. In spite of these problematic issues, portions of 
the Model IV Intervention were successfully implemented. After a slow start, school 
improvement specialists were able to gain the trust of principals and staff, provide guidance in 
data-driven decision making, help the school improvement planning committees develop and 
maintain the SIPs, and elicit increased collaboration between central office staff and school 
faculty. 



Because the Petersburg City School Division has a high staff turnover rate, school 
capacity for improvement is always in flux, as are leadership quality, teacher quality, and 
curriculum continuity. To help address these issues, the Virginia Department of Education made 
available an extra $1 million to be used as signing bonuses in hiring qualified teachers, with the 
stipulation that these newly hired teachers must commit to remaining in the division for 3 years. 

Recommendations 

The school division should consider the following steps to sustain gains and support 
continued improvement: 

• Institutionalize the central office school improvement specialist and ensure the 
continuance of the COST program. This might be accomplished by creating a 
permanent position or by assigning the specialist’s responsibilities to a central office 
staff member who has the necessary qualifications. 

• Call upon (and support) each school’s instructional specialist, to whom the school 
improvement specialists provided valuable data analysis support and instruction, to 
provide instruction in data analysis and to perform data analysis as needed. 

• Set target dates for achieving school improvement objectives. Instead of requiring 
schools to implement the entire SIP at once, the SIP might better be administered in 
an easily managed order, predetermined either at the central office level or at the 
individual school level and supported by the central office. 

• Act to stabilize the teaching and leadership staffs. The recently approved signing 
bonuses are a positive move toward addressing the faculty turnover rates in the 
division. Moving toward stability in teaching and leadership staffs may provide the 
necessary continuity to implement multiyear reform efforts successfully. 
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Introduction 



In its efforts to improve student achievement, the Virginia Department of Education is 
interested in determining best practices for supporting low-performing schools. “Low 
performing” refers to those schools that do not meet the standards established and monitored by 
the state board of education, which has no legal authority to take over failing schools. The 
Petersburg City Schools Division was selected as a test site for one of the four model 
interventions because its seven elementary schools, two middle schools, and one high school 
have a history of poor student achievement . 1 

Dr. Jo Lynne DeMary, Virginia’s state superintendent of public instruction, requested 
that the Appalachia Educational Laboratory at Edvantia work in partnership with the Virginia 
Department of Education and Petersburg City Schools to design and test the Partnership for 
Achieving Successful Schools Initiative (PA+SS) Model IV Intervention. The goal was to help 
the schools in Petersburg move toward full accreditation in the state’s accountability system. 
Edvantia selected The CNA Corporation (CNAC), a private, nonprofit research and evaluation 
organization, to conduct an independent but coordinated evaluation of the PA+SS Model IV 
Intervention project in Petersburg City Schools. 

Model IV Intervention is one of four models now being tested by the Virginia 
Department of Education as part of the PA+SS project. Each model provides technical 
assistance, but each takes a different approach depending on the level of need in the school or 
school system. Model I utilizes a state team of educators who conduct academic reviews at the 
school level. The team looks for evidence of best practices, uses of data, curriculum alignment, 
and professional development. Team members review test scores, conduct classroom 
observations, and interview principals. Areas identified for improvement become “Essential 
Actions” that are formulated as recommendations and incorporated into the school improvement 
plan (SIP) soon after the visit. After the initial school visit, the academic review team follows up 
at each school through a series of visits to determine whether implementation of the Essential 
Actions was accomplished. The Model II intervention functions in the same way, with one 
exception: an external teacher and principal make monthly visits to schools that are accredited 
with warning. The Model III intervention involves all the elements of Models I and II but also 
includes assistance from an external principal, trained in school improvement theory and 
strategies, who is assigned to work one week per month with the principal and faculty in each 
school accredited with warning. 

The Model IV Intervention, unlike other models, involves working at the school 
division's central office. It calls for placing a school improvement specialist in each school and 
another school improvement specialist in the central office (referred to as the central office 
specialist). The intervention is the most intensive of the four models and is aligned with the state 
requirement that a school division have an improvement plan when there is little improvement in 
student achievement. 



1 The Petersburg school division also includes two alternative schools. They are not included in this evaluation. 




Purpose of This Report 

This report presents the evaluation findings for the Model IV Intervention implemented 
in the Petersburg City School Division during Years 1, 2, and 3 (school years 2002-2003 through 
2004-2005) of the 3 -year divisionwide school improvement project. The report presents an 
analysis of the fidelity with which the intervention was implemented. It explores the role and 
effectiveness of the school improvement specialists, the collaboration among the division’s 
central office and the schools and school improvement specialists, the completeness of individual 
SIPs, and the changes that have occurred in the school’s capacity for improvement and in the 
faculty’s perceptions of themselves as continuously improving. This report also presents a 
comparative analysis of the percentages of students passing the Standards of Learning (SOL) 
tests in Petersburg and Richmond City Schools during the four years prior to the Model IV 
Intervention and the two years for which data are available since the intervention was initiated. 

Model IV Intervention 

During the 2002-2003 school year, the Virginia Department of Education and the 
Appalachia Educational Laboratory at Edvantia began implementing Model IV of the PA+SS 
project in Petersburg City Schools. The implementation continued through the 2004-2005 school 
year and was completed in the spring of 2005. The intervention provides a system of external 
facilitation. The long-term goal is to improve student achievement to levels that ensure full 
accreditation in compliance with the state accountability model. In the short term, the 
intervention aims to ensure that 

• data are used to make sound instructional decisions at both the school and district 
levels 

• school and district faculty and staff have the skills needed to develop strategic 
improvement plans based on these data 

• structures are in place and used for monitoring the fidelity with which the strategic 
plan is being implemented 

Model IV focuses school improvement planning in specific skills, concepts, and content 
by funding school improvement specialists to serve as external facilitators in each school that is 
accredited with warning. A warning status indicates that the school has not fully met state 
accreditation standards, particularly those related to student achievement. To help facilitate 
school improvement, the school improvement specialist is a member of the school improvement 
committee and has direct input into the development of the school improvement plan (SIP). The 
Model IV Intervention also features a central office specialist who serves as an external 
facilitator at the central office level to ensure coordination of the individual school improvement 
plans with the division improvement initiatives. Following is a more detailed description of the 
three main features of the Model IV Intervention. 

School improvement specialists. School improvement specialists are assigned to schools 
to provide support by guiding school personnel in assessing needs and developing strategic 
improvement plans. Each of the school-based school improvement specialists is an experienced, 
retired educator recommended by the Virginia Department of Education because he or she has 
successfully worked with low-performing schools. All receive training from Edvantia that 
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includes use of data for decision making, school improvement planning, curriculum alignment, 
and monitoring instruction. The specialists also receive in-depth training in facilitation skills. 

The length of training sessions varied — some were held after school, some were held on 
teacher workdays, and others were held as preschool sessions. During the first year there were 
five 2-hour training sessions held after school and on teacher workdays. During the second year, 
school improvement specialists attended a 3-day workshop on writing school improvement plans 
and attended five to eight after-school and teacher workday sessions. During the third year, 
school improvement specialists attended a weeklong staff development seminar held for 6 hours 
each day. In addition, school improvement specialists attended five to eight after-school and 
teacher workday follow-up staff development workshops. The specialists met with each other 
monthly to reflect on their experiences in the schools and to receive additional training as 
needed. 



The school improvement specialists played a role in assessing readiness for change and 
worked with the school leadership team to build a culture that is supportive, sustainable, and 
focused on improving student achievement. They were assigned to work on-site at a school at 
least one day each week. The role of the school improvement specialist was to 

• support the principal, school leadership team, and teachers as they used data to 
determine the school’s needs related to improving student achievement 

• provide consultation, support, and direct service to the principal and school faculty as 
they developed a school improvement plan with goals, objectives, and strategies 
targeted to the areas of need found through data analysis 

• provide support to the principal and faculty to implement the school improvement 
plan and monitor the success of the strategies 

Central office specialist. Originally, all schools in the Petersburg City Schools Division 
were accredited with warning. Therefore, a school improvement specialist, referred to as the 
central office specialist, was assigned to work with the central administration for up to 20 hours 
each week. Because more intense central office support was required to accomplish the goals of 
the intervention, the central office specialist role expanded over time to four days per week. The 
role of the central office school improvement specialist was to 

• coordinate and monitor the activities of the school improvement specialists at each 
school 

• provide consultation and support to the central office administration to ensure that 
policies and practices supported the work of the schools in improving student 
achievement 

• support the superintendent, central office leadership team, and school board as they 
used data to determine the school district’s needs related to improving student 
achievement 

• support the development of a division improvement plan on which individual schools 
built their improvement efforts 

• build structures and processes to support central office monitoring of the 
implementation of the improvement plans 
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• link the school division to additional resources, such as those offered by the Virginia 
Department of Education, for assistance in the improvement process 

The improvement specialists at the school and central office levels worked to build the 
school community’s capacity to use data to develop and implement a school improvement plan. 
The specialists provided training in data analysis and strategic planning and modeled the trained 
behaviors. 

School improvement plan. A key feature of the Model IV Intervention is the school 
improvement plan (SIP). The school improvement specialist assigned to each school supported 
the principal and faculty by including jump-start strategies in the school improvement plan to 
move the school forward quickly." These strategies include the following: 

• aligning the written curriculum with the taught curriculum 

• aligning local and classroom assessments with the curriculum and continuously 
monitoring student progress 

• analyzing student achievement data, including disaggregated data for groups such as 
minorities and special education students, to identify the most critical needs for which 
immediate, common achievement goals and strategies might be developed 

• putting structures in place to monitor both delivery of the instructional program and 
student progress toward meeting the achievement goals 

• providing additional learning time for students 

To maintain the momentum gained through the jump-start strategies, each school 
improvement specialist supported his or her assigned school principal and faculty to address 
organizational and cultural changes needed to sustain improved student achievement. Through 
standardized surveys of school culture, as well as interaction with faculty, administration, and 
community, the school improvement specialist helped to determine the readiness of the faculty 
and the community to change practices, use data in planning for change, foster faculty 
collaboration, and distribute leadership functions. School improvement specialists influenced and 
shaped these strategies through participation on their assigned school’s improvement committee. 

Target Group 

The Petersburg City School Division is an urban school division serving 5,400 students, 
from kindergarten through 12th grade (K-12). More than 85% of the students in the division 
qualify for free or reduced-price lunch. All seven elementary schools and both middle schools 
are designated as Title I schools (the high school is not). 

Evaluation Plan 

The main goals of this study are to document implementation of the Model IV 
Intervention in Petersburg City Schools and to determine what difference it made in student 
achievement. The core research questions are as follows: 



2 For more information about the jump-start strategies, see the short-term strategies in Corallo, C., & McDonald, D. 
(2003). What works with low-performing schools. Charleston, WV: AEL. 
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• Goal 1: To what extent was the Model IV Intervention plan implemented with 
fidelity? 

• Goal 2: How successful was the Model IV Intervention in improving student 
achievement? 

To address these two broad goals, the overall study seeks to answer four specific research 
questions. 

1. What evidence is there that the intervention affected the quality of the school 
improvement plan? 

School improvement plans cover a period of 3 years and are updated annually. The 
school improvement committee takes responsibility for writing the plan, conducting monthly 
reviews, and updating the plans annually. Researchers used a rubric, the Virginia School-Level 
Academic Review School Improvement Plan, Data Collection Summary Sheet — shown in 
Appendix A — to assess the presence of school improvement plan indicators relating to (1) 
content plan strategies, (2) implementation strategies, (3) results strategies, and (4) jump-start 
strategies. 

2. What evidence is there that the intervention was implemented according to the Model 
IV Intervention plan? 

Evaluators conducted interviews with school improvement specialists in spring 2004 and 
2005. They also collected, read, and analyzed each school improvement specialist’s 2-year 
compilation of weekly field notes. Evaluators also conducted spring 2004 and 2005 interviews 
with the central office specialist, school leaders, and central office leaders with whom school 
improvement specialists interacted during the study period. The interview data and specialist 
field note journals were used as a means of providing insight into the implementation of the 
intervention. 

3. Do school culture and/or instructional quality change when the intervention is 
implemented? 

a. What were some major changes in teacher collaboration? 

b. What role did the school improvement specialists play in guiding school 
leadership? 

The research question posed was addressed by the interviews with school improvement 
specialists and analysis of their weekly field note journals. In addition, faculty members’ 
continuous improvement (as they perceive it) and the school’s capacity for improvement were 
measured using the Continuous School Improvement Questionnaire (CSIQ) and Measure of 
School Capacity for Improvement (MSCI). The CSIQ and MSCI are surveys administered 
annually to all Petersburg school professional staff. 

4. Has there been growth in achievement as measured by SOL tests? 
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Growth in student achievement is measured by school-level SOL data collected for the 10 
Petersburg City schools and their matched counterparts in Richmond City. SOL tests are given 
annually in Grade 3 for English, math, history, and science; in Grades 5 and 8 for English, 
writing, math, history, and science; and in high school for English, writing, math, science, and 
history courses. 

The evaluators used a quasi-experimental, matched comparison group design. For 
comparative purposes, student achievement data were collected in all 10 Petersburg City schools 
and in one-to-one matched comparison group schools in Richmond City, yielding a sample size 
of 10 matched pairs. 3 Matches were made based on several characteristics, including the size of 
the school, grade levels, poverty indicators, and available SOL test results. 

In general, this evaluation report focuses on the following issues: 

• patterns and trends in SOL results for Petersburg schools in comparison to Richmond 
City schools 

• the role of the school improvement specialists 

• the merits of the school improvement plans 

• changes in perceptions of the schools as high-performing learning communities and 
in the schools’ capacity for improvement 



3 This Model IV Intervention was not compared to Models I-III because schools and divisions were assigned to the 
models based on need; therefore, schools assigned to the other models are not comparable to the Petersburg 
schools. In addition, the intervention and research teams did not have access to data from schools using the other 
models. 
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Standards of Learning 



The Standards of Learning (SOLs) for Virginia public schools define the 
Commonwealth’s expectations for student learning and achievement in Grades K-12. SOL tests 
cover English, mathematics, science, and history/social science. The Virginia Department of 
Education Web site indicates that the content of the SOLs represents a consensus of what 
parents, classroom teachers, school administrators, academics, and business and community 
leaders believe schools should teach and students should master. In the four core areas (English, 
mathematics, science, and history/social science), the Commonwealth provides teachers with 
curriculum frameworks that detail the specific knowledge and skills students must demonstrate 
to meet standards of learning for these subjects. 4 For each school, SOL end-of-course test results 
are reported by the state in terms of the percentages of students passing (i.e., scoring at proficient 
or advanced) on each subject area test. These pass rates are available from the Virginia 
Department of Education Web site. 5 

Analysis of SOL Results 

To evaluate the impact of the Model IV Intervention program, the 10 Petersburg City 
schools were paired with similar schools in the city of Richmond. The Richmond City schools 
are the comparison group for the Petersburg schools (i.e., because there is no intervention 
program in the Richmond City schools). CNAC developed the school-matching criteria used to 
select schools in Richmond City Schools for the comparison group. 

SOL pass rates at the school level were downloaded from the Virginia Department of 
Education Web site. Again, the SOL results show the percentages of students passing the SOL 
tests by grade level and subject. Data used for this evaluation cover the 5 years prior to the 
school improvement intervention (1998, 1999, 2000, 2001, and 2002) and the first 2 years of the 
3-year intervention period (2003 and 2004). Data for 2005 were not yet available at the time this 
report was written. Evaluators were thus able to compare SOL results for the baseline years 
(1998 to 2002) with the intervention years (2003 and 2004) in areas of student achievement 
targeted by the Petersburg City Schools’ school improvement plans. The SOL pass rates are 
shown in Appendix F for (1) third- grade English, mathematics, history, and science; (2) fifth- 
and eighth-grade English, writing, mathematics, history, and science; and (3) high school 
English, writing, Algebra II, geometry, U.S. History I and II, World History I and II, world 
geography, earth science, biology, and chemistry. The evaluators’ purpose for collecting these 
data was to compare trends in student performance across the years between each Petersburg 
school and its matched comparison school. 

Table 1 shows the number of Petersburg and comparison schools that have improved 
their SOL subject pass rates by grade level and subject. Also shown are increases (or decreases) 
in pass rates, by grade level and subject, for Petersburg and comparison schools; increases (or 
decreases) in the state averages are also shown. 



4 www.pen.kl2.va.us/VDOE/Superintendent/Sols/home.shtml 

5 www.pen.kl2.va.us/VDOE/Assessment/2002SOLpassrates.html 
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Table 1. Increases (or Decreases) in the Percentages of Students Passing SOL Tests Before 
(1998-2002) and After (2003-2004) the Intervention: Petersburg Schools, Comparison Schools, 
and the State Average 



Grades and 
Subjects Tested 


Petersburg Schools 


Comparison Schools 


State Average 


Number 

Improved 


Increase in 
Percentage of 
Students 
Passing 


Number 

Improved 


Increase in 
Percentage of 
Students 
Passing 


Increase in 
Percentage of 
Students 
Passing 


Third Grade 












English 


3 


-1 


4 


11 


-1 


Math 


6 


18 


7 


16 


7 


History 


6 


15 


6 


13 


11 


Science 


7 


18 


5 


13 


8 


Fifth Grade 












Writing 


5 


10 


7 


39 


4 


English 


7 


15 


7 


45 


7 


Math 


7 


27 


6 


30 


7 


History 


6 


30 


7 


46 


14 


Science 


7 


33 


7 


51 


8 


Eighth Grade 












Writing 


1 


4 


2 


27 


1 


English 


1 


-7 


2 


8 


1 


Math 


2 


13 


2 


25 


7 


History 


2 


19 


2 


27 


5 


Science 


2 


4 


2 


10 


3 


High School 












Writing 


1 


1 


1 


8 


1 


English 


0 


-6 


0 


-1 


3 


Algebra I 


1 


30 


1 


19 


2 


Geometry 


1 


5 


1 


39 


6 


Algebra II 


1 


25 


1 


31 


9 


U.S. History 


1 


19 


1 


8 


15 


World 
History I 


0 


-26 


NA 




-3 


World 
History II 


0 


-2 


1 


4 


4 


Earth 

Science 


1 


10 


1 


10 


1 


Biology 


0 


-17 


1 


22 


-3 


Chemistry 


1 


25 


1 


15 


8 



In the Petersburg division, the pass rates for writing and English decline for some grade 
levels over the course of the intervention. In the grades where it does not decline, the increases in 
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the percentages of students passing were smaller than in the comparison schools or in the state 
overall. Math gains show a more positive picture. Although gains were not always as impressive 
in Petersburg as in the comparison schools, the gains made in math were above the state average. 
Petersburg High School saw a large increase in the number of students passing the SOLs for 
Algebra I and II, U.S. history, and chemistry. Pass rates in world history and biology declined in 
Petersburg while the comparison school made gains. 

Appendix B provides a series of charts to help the reader visualize trends in the SOL 
scores and to assist in determining the effectiveness to date of the Model IV Intervention 
program for the first and second years of the intervention. The data used in these charts represent 
an average of the scores for the Petersburg schools, the Richmond comparison schools, and all 
schools in the state for each SOL test and year combination. 

For example, see the figure, which displays the average fifth-grade science SOL pass 
rates for all 7 elementary schools in the Petersburg school system, for the 7 comparison schools 
in Richmond, and for the state overall. A least-squares linear trend (1998 to 2004) is plotted for 
the Richmond comparison elementary schools, the state elementary school average, and the 
Petersburg elementary schools. For the Petersburg schools, a least-squares linear trend is plotted 
for the pre-intervention period (1998 to 2002) and for intervention years (2003 and 2004). 



Fifth Grade Science 




Year 



Figure. Fifth-Grade Science SOL Test Trends for Petersburg Schools, Richmond 
Comparison Schools, and the State Average. 

The trend lines in the figure show that, for the Richmond comparison schools, there has 
been a steady improvement in student pass rate averages for the fifth-grade science SOL test. 



9 



The coefficient of determination (i.e., /^-square statistic) for the Richmond schools’ trend line is 
0.82, suggesting a moderate to high degree of correlation between the actual scores and the linear 
trend line. However, because these trends are not being used to forecast scores but instead to 
visualize trends — pre-intervention and during intervention — the focus is on the relative slopes of 
the trends and the extent to which the Petersburg schools exhibit a marked improvement in the 
scores during the first 2 years of the Model IV Intervention period. Accordingly, for this SOL 
test (fifth-grade science), the Petersburg schools have demonstrated a marked improvement in 
the pass rates, and the trend in improvements is markedly higher during the intervention period 
than during the pre-intervention period (as demonstrated by the increased slope of the 
intervention period trend line for Petersburg schools). Although Petersburg’s and Richmond’s 
pass rates in fifth-grade science are rising more quickly than the state average, both school 
divisions’ pass rates fall well below the state average. 

Charts for each of the SOL subject tests averaged for grades (except high school) are 
presented in Appendix C. Findings may be summarized as follows: 

• Third-grade SOL tests: The difference in achievement between Petersburg schools 
and Richmond comparison schools appears to be lessening, with Petersburg schools 
making larger gains and, hence, approximating the achievement of the Richmond 
comparison schools. 

• Fifth-grade SOL tests: Results on all tests are trending up. 

• Eighth-grade SOL tests: Writing, math, science, and history pass rates for Petersburg 
schools were generally lower than the Richmond comparison schools during the pre- 
intervention period; however, notable gains have been made, and the differences 
between Petersburg schools and the Richmond schools are trending up, with the gap 
becoming comparatively smaller for the history SOL than for the writing, math, or 
science SOLs. The Petersburg schools’ English 2004 SOL pass rate has improved 
slightly over the English 2003 SOL pass rate, but it is still lower than the SOL pass- 
rate score achieved in the pre-intervention year (2002). 

• High school SOL tests: Petersburg pass rates were generally higher than the 
Richmond comparison school pass rates during the pre-intervention period; however, 
Petersburg pass rates were generally lower than Richmond’s by the second year of the 
intervention period, with the exception of writing, Algebra I, and U.S. history. Both 
high schools (Petersburg and Richmond) made gains in all subject areas during the 
first year of the intervention period. However, Petersburg achievement is not 
increasing at a greater rate than Richmond City. This trend is more evident by the 
second year of the intervention. 

Summary of SOL Findings 

Although analyses of SOL findings are tentative due to the unavailability of the 2005 
scores, the trend appears to show that the Petersburg schools, having started with very low pass 
rates, are improving over time at a faster rate than comparison schools. Not all grades and 
subject areas are improving at the same rate, but notable improvements in pass rates are evident. 
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Role of the School Improvement Specialists 



In examining the role of the school improvement specialists, the evaluators drew upon 
three main sources of data collected during for the 3-years of the intervention: (1) the school 
improvement specialists’ monthly contractor reports (2003-2005) and invoices (2003-2004), 

(2) their weekly field notes (2003-2005), and (3) interviews with the school improvement 
specialists and the school leaders with whom they interacted. For the central office specialist, 
weekly field notes for the 2002-2003 year and interview data for 2003-2005 were examined. 

Data Sources 

The contractor reports and invoices provide a summary of school improvement 
specialists’ activities and level of effort (number of days worked per month). The weekly 
journals provide insight into the intervention and its evolution. The interviews, conducted at the 
end of the second and third years of the initiative, present a more distilled view of the process 
and one with additional dimensions, because school personnel — particularly principals — in 
addition to school improvement specialists were interviewed. Combining these data makes it 
possible to view the same event from several perspectives and to develop a rich understanding of 
the situations faced by these key facilitators of the PA+SS initiative and their clients. 

During the first 2 years of the intervention several themes emerged. Most are problems 
the school improvement specialists encountered in their work, and others are barriers that made 
implementation of PA+SS Model IV difficult. These themes include 

• lack of trust between school improvement specialists and school personnel 

• complex and confusing leadership dynamics at several levels 

• micromanagement by school board officers 

• high turnover of teachers and administrators 

• concerns about the quality of teachers and administrators 

• need for teacher professional development 

• discipline and classroom management problems 

• poor coordination of efforts between schools and central office 

• too many new initiatives and not enough follow-through 

While many of the themes were observed again in Year 3, there were some changes. 

Most notable were 

• improvements in school improvement specialists’ relationships with school leaders 

• greater trust between school leaders and staff 

• an increase in community support 

• a decrease in micromanagement by the school board 

• better coordination of efforts by the Central Office Support Team 

• professional staff’s perceptions of themselves as continuously improving throughout 
the implementation process 
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These themes are discussed later in this section. First, the data sources are described in 
some detail. 

Monthly contractor report and invoice. In the monthly contractor report and invoice, 
the school improvement specialists recorded the number of days they worked each month and 
briefly described the nature of their contributions. The evaluators did not have complete records 
for all of the school improvement specialists, but available records allowed them to determine 
the level of effort typical of a school improvement specialist. 

Interviews. In summer and fall of 2004 and 2005, the study team conducted telephone 
interviews with 7 school improvement specialists assigned to the schools, 6 the central office 
specialist, the principals at 8 of the Petersburg schools, another administrator at the 2 schools for 
which the principal was not available, 1 central office administrator, and 1 board member. These 
interviews used a standard set of questions aimed at eliciting information about the intervention 
and its perceived success. The protocols for the school improvement specialists and Petersburg 
school personnel were slightly different to account for the different perspectives of the two 
groups. The interviews covered the following topics: 

• school improvement specialists’ perceptions of the schools 

• school staff s perceptions of the school improvement specialists 

• the role of the school improvement specialists 

• pressing needs of the schools 

• whether and how the PA+SS Model IV Intervention addressed those needs 

• school improvement specialists’ accomplishments 

• challenges and disappointments 

• lessons learned 

• effectiveness of the school improvement planning process 

• AEL/Edvantia’s performance 

• perceptions of the impact of the program 

School improvement specialist field notes. School improvement specialists were asked 
to keep field notes on their activities. They were to document not only what they did, but also to 
reflect on and record what they had learned and to record their perceptions of their effectiveness. 
The journals vary in approach and content. Several make compelling reading, with lengthy notes 
detailing the ups and downs of the school improvement specialist’s experiences and the reception 
he or she received from administrators and teachers. One is a terse list of activities with limited 
commentary. The others fall between these two extremes. Because of the variation in style and 
the quantity of information, it is not possible to gain a comprehensive picture of the intervention 
across the entire division from this source. To preserve anonymity, individual school 
improvement specialists or their school assignments are not identified in this report. 

School Improvement Specialist Activities, Overall 

This section covers the school improvement specialists’ activities. This picture of what 
the school improvement specialists did is constructed primarily from the monthly contractor 

6 Several school improvement specialists worked with more than one school. 
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